Papers with lightweight expansions
SCALE: Upscaled Continual Learning of Large Language Models (2026.findings-acl)
Copied to clipboard
Jin-woo Lee, Junhwa Choi, Bongkyu Hwang, Jinho Choo, Bogun Kim, Jeongseon Yi, Joonseok Lee, DongYoung Jung, Jaeseon Park, Kyoungwon Park, Suk-hoon Jung
| Challenge: | Recent discussions suggest that further progress will come from scaling the right structure, not merely parameters or data, while preserving acquired knowledge. |
| Approach: | They propose a width upscaling architecture that inserts lightweight expansions into linear modules while freezing all pre-trained parameters. |
| Outcome: | The proposed architecture reduces severe forgetting while learning new knowledge on a controlled synthetic biography benchmark. |